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A STATISTICAL TEST FOR MEANS OF SAMPLES 
FROM SKEW POPULATIONS 


LEON FESTINGER 


IOWA CHILD WELFARE RESEARCH STATION 
STATE UNIVERSITY OF IOWA 


This paper presents a test for determining significance of dif- 
ferences between means of samples which are drawn from positively 
skewed populations, more specifically, those having a Pearson Type 
III distribution function. The quantity 2npx,/x, (where p equals 


the mean squared divided by the variance and m is the number of 
cases in the sample), which distributes itself as Chi Square for 2np 
degrees of freedom, may be referred to the tables of Chi Square for 
testing hypotheses about the value of the true mean. For two inde- 
pendent samples, the larger mean divided by the smaller mean, 
which distributes itself as F for 2n,p, and 2n,p, degrees of freedom, 


may be referred to the F' distribution tables for testing significance 
of difference between means. The test assumes that the range of 
possible scores is from zero to infinity. When a lower theoretical 
score limit (c) exists which is not zero, the quantity (Mean — c) 
should be used instead of the mean in all calculations. 


Exact tests of significance for means are known when the sam- 
ples are drawn from normally distributed populations. In a previous 
article (2) a method has been described for testing significance of 
means when the samples come from exponentially distributed popu- 
lations (J curves.) Between these two lies a range of skew frequency 
distributions for which we as yet have no statistical tests of signifi- 
cance. It is the purpose of this article to generalize the test for ex- 
ponential distributions to all skew distributions of Pearson’s Type 
III, of which the exponential distribution is a special case. The types 
of distributions to which this test may be applied can be seen in Fig. 
1, where are plotted four Pearson Type III curves ranging from 
marked skewness to moderate skewness. 


Derivations 


Let us write the distribution function with which we shall deal 
in a general form 
f(z) =carte’; OS 2<%. (1) 
If we now set 
c fea e*dr=k, (2) 


205 


= 


See 


SSeS 


ene s e 


2g 











206 PSYCHOMETRIKA 

















F(X) 

41 
pel 
pre 
prs 

ci 

7) 

rT p-/0 

37 e 

27 

J4 

oO 





or,@e2se¢SB C6@vreoehm Ww x 
FIGURE 1 


Examples of Pearson Type III distributions of varying skewness with x = 8. 


where k is the total area under the curve, and also set 
C [» e- br dx=2Z,, (3) 
k J, 


where x, is the arithmetic mean of the distribution, we find after 
evaluating the integral that 


cl’ 
(p) _, and a 
b’ Ly 





We may therefore rewrite equation (1) so that the constants are ex- 
pressed in terms of the moments of the distribution: 
k(p)? 


f(2) =—————— ae ey): OS 4m: (4) 
x I'(p) 


Let us now set the area (k) equal to unity. 

















G. R. THORNTON 207 


Craig (1) and Irwin (3) have both, by different methods, ar- 
rived at the distribution function of the means (x,) of samples of size 
n drawn from the population of equation (4): 





(np)”? 
p (Xz) = ——_—— 9,,"9-1 @-(mpta/2p) | (5) 
cd I' (np) 
By rearrangement of terms we may write this in a more convenient 
form: 
4 np-1 
(=) ) . E- ("Pt s/Lp) 
aie +» (5a) 
p(X.) = : 
I’ (np) 


Similarly to the procedure described in connection with the ex- 
ponential populations (2), we may now reduce this to a Chi Square 
distribution by making the substitution 














2np xs 
oo ‘ (6) 
S 
and of course 
Xp 
dx, = : dz. (7) 
np 
Then, setting 
*2e2 * (Lp/2np)z2 i 
| f(x,)dz,= | f(z)dz, (8) 
2a1 (Lp/2np) 23 
we obtain 
g np-1 bi 
a e-(*/2) 
F(z) = =7(="2) (9) 
2 I'(np) fy 


which is the distribution of Chi Square for 2np degrees of freedom. 

Thus the quantity 2npz,/x, may be referred to the table of Chi 
Square for the appropriate number of degrees of freedom for the pur- 
pose of determining the fiducial limits of the true mean for any ob- 
tained sample mean. This quantity, it may be observed, is identical 
with the quantity used for the exponential populations when p equals 
1, under which circumstances equation (4) reduces to an exponential 
distribution. It is obvious that p is a measure of the skewness of the 
distribution; the larger p is, the less skew is the distribution. 

It is interesting to note the different results obtained between 
this test and the application of a ‘t’ test to distributions of this type. 








208 PSYCHOMETRIKA 


This is shown in Fig. 2, where the 2 per cent fiducial limits have been 
calculated for the case of the obtained sample mean equal to 1, and 
the number of cases in the sample equal to 9 for » varying from 1 
to 30. 


— from "ft" distribution 
---from x® distribution 
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FIGURE 2 


Comparison of 2 per cent fiducial limits determined by ‘t’ test and Chi Square 
test for samples of x = 1, = 9. 


In order to apply the ‘t’ test it is necessary to know the variance 
of the sample. If we set 


i J “a et(x — x,)*de=er, (10) 
we find 
1 
o? =-2,". (11) 
p 


Using the right-hand quantity of equation (11) we obtained the fidu- 
cial limits consequent to applying a ‘t’ test as shown by the broken 
lines in Fig. 2. The solid lines show the fiducial limits obtained from 
referring 2npx,/x, to the Chi Square table. As is to be saline, the 
more skew the population, the greater the error introduced by the 
use of the ‘t’ test. For p greater than 15, for this number of cases in 
the sample, there is little difference between the results obtained from 
the two tests. Thus, if the population is only slightly skew the ‘t’ test 
can be used without appreciable error. For markedly skew popula- 
tions, however, the ‘t’ test is not permissible and the correct test de- 
scribed in this paper should be used. 

For the test of significance between means we may apply the 
same reasoning which we applied previously (2). Since 2npz,/x, is 
distributed as Chi Square for 2np degrees of freedom, and since 2, 
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is an unbiased estimate of x,, then, assuming independence of 2%; 
and 2.2, 
Lei 


—=F (for 2n.p, and 2n.p, degrees of freedom) . (12) 
Xs2 


Thus the ratio of the larger to the smaller mean may be referred to 
the table of F for the appropriate number of degrees of freedom. 

It is, of course, clear that before we can refer our statistics to 
either the table of Chi Square or the table of F with the appropriate 
number of degrees of freedom, we must determine the value of p. 
Referring back to equation (11), we see that p may be evaluated sim- 
ply by dividing the mean squared by the variance. To be absolutely 
correct, the sampling distribution of p should be taken into account 
in our formulas. This is not, however, a serious objection since the 
sampling distributions of Mean and Sigma are positively correlated 
and the variance of the sampling distribution of p will, therefore, in 
any case, be quite small. Very little error will therefore be introduced 
by ignoring the sampling distribution of p. 

We shall not present any concrete applications of the test at 
present, since several illustrations are given in the previous paper 
(2) and the method of analysis is identical. We may, however, sum- 
marize by reviewing the exact method of applying the present test. 


Summary 


The test applies to samples drawn from populations with a Pear- 
son Type III frequency distribution. Most skew distributions ob- 
tained in practice will probably be adequately represented by a Pear- 
son Type III curve. In cases of doubt a Chi Square distribution of 
2p degrees of freedom may be fitted to the distribution of 2px7/x, and 
tested for goodness of fit. 

When the test is applied to the difference between two means, 
the two samples are assumed to be independent. 

If the lowest possible score is some constant ‘c’ greater than zero, 
the quantity used in application of the test should be (x, — ¢). 

To find the fiducial limits of the true mean, the two values of Chi 
Square at the desired level of significance times the true mean are 
each in turn set equal to 2npzx, . 

To test the significance of difference between means, equation 
(12) is used. In calculating this ratio the larger mean should always 
be used as the numerator. 
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The value of » is determined by equation (11). 
In cases where p is large, say over 10, and the number of cases 


in the samples is appreciable, say greater than 15, the ‘t’ test may be 
used without much error. 
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THE SIGNIFICANCE OF RANK DIFFERENCE 
COEFFICIENTS OF CORRELATION 


G. R. THORNTON 
PURDUE UNIVERSITY 


The coefficients of rank difference correlation that are barely 
significant at six different levels of significance are given for N’s 
of 2 to 30. Most of the values were obtained by translation of Olds’ 
tables of probabilities for various values of =d?. Comparison of 
these data with those obtained by four other methods indicates that 
one method yields values more appropriate than those obtained from 
Olds’ data for coefficients significant at the .01 level for N’s from 11 
to 25. This method also provides a convenient means of obtaining 
approximate values of coefficients significant at the .01 level for N’s 
above 80. Need for caution in evaluating the significance of coeffici- 
ents obtained from data involving tie rankings is inglicated. The ar- 
ticle concludes with recommendations as to choice of methods of de- 
termining the significance of rank difference coefficients. 


Until recently there has been no satisfactory method for testing 
the significance of a coefficient of correlation (7’) obtained by Spear- 
man’s rank difference method by the formula, 


6 
r=1-— ae, (1) 
N? — N 

In a recent paper E. G. Olds (5), extending the work of Hotelling and 
Pabst (3), presents tables giving the probabilities for various values 
of Sd? for N’s of 2 to 30. These tables should be very useful to persons 
who wish to test the significance of coefficients while in the process of 
calculating them by the formula given above. 

A translation of Olds’ tables into a table giving probabilities for 
the coefficients themselves seems desirable for two reasons: (1) Meth- 
ods of calculating a rank difference coefficient of correlation are now 
in use which do not require the calculation of Sd? (1). (2) Olds’ 
tables are not readily applicable to the interpretation of rank differ- 
ence coefficients reported in the past or present psychological litera- 
ture. 

Table 1 presents a translation of Olds’ probabilities into rank 
difference coefficients of correlation which are barely significant at 
each of several levels of significance for N’s of 2 to 30. The prob- 
abilities listed at the head of the columns apply to both positive and 
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Coefficients of Rank Difference Correlation that are Barely 
Significant at Given Levels of Significance 
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TABLE 1 














N 01 .02 .04 05+ 10 .20 
2 none none none none none none 
3 none none none none none none 
4 none none none none O17 .883 
5 none .989 .948 .933 .861 .729 
6 .960 .928 .868 .843 .774 .636 
7 .906 .867 .806 -781 .695 571 
8 .869 .816 .749 -725 .634 514 
9 .825 TH -705 .681 .592 ATT 

10 -788 -733 .668 .644 .555 -445 
11 816 (.764)* -736 .651 .626 521 406 
12 117 (.786) -702 .620 .599 496 387 
13 -145 (.71 .672 594 .575 .476 3871 
14 -720 (.68 .646 571 .553 457 356 
15 689 (.667) 623 .550 5385 .441 3843 
16 .666 (.648) -602 531 517 426 332 
17 645 (.630) 583 .514 .501 .412 321 
18 .626 (.614) .565 499 487 .400 012 
19 608 (.598) .549 485 AT4 889 3038 
20 592 (.583) .5385 472 462 378 .295 
21 577 = (.571) 521 .460 .450 369 -288 
22 563 (.558) 509 .449 .440 .360 .281 
23 550 (.547) 497 489 430 352 274 
24 588 (.536) 486 429 .420 344 .268 
25 527 (.525) .476 .420 .412 837 -263 
26 .516 .466 .412 .404 3830 257 
27 .506 457 404 396 324 .252 
28 497 449 396 389 018 .248 
29 488 441 389 382 012 .243 
30 479 .433 382 875 806 239 








* The values given in parentheses are preferable. 


(See text below.) 


+ The values given for W’s of 11 to 30 in the .05 column are not so accurate as other values 
in the table. Their derivation is explained below. 
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negative coefficients. The coefficient .960 appearing opposite an N of 
6 in the .01 column, for example, means that by chance a coefficient 
as large as .960 or larger may be expected to occur once in 200 times 
in the positive direction and once in 200 times in the negative direc- 
tion. Where “none” appears in the table, it is to be interpreted to 
mean that no coefficient is significant at that level for the given N . 

To obtain the coefficients in Table 1 from Olds’ data several ad- 
justments were necessary. For N’s from 11 to 30 Olds presents limits 
of >d? within which 99 per cent, 98 per cent, etc., of the cases will 
occur by chance. To obtain the figures in Table 1 for N’s from 11 to 
30, inclusive, I have merely calculated the coefficients of correlation 
for the limits given by Olds and then added to each .001, with a few 
exceptions. The exceptions include the following: al) coefficients for 
N of 12; the coefficient at the .01 level for N of 14; the coefficient at 
the .02 level for N of 29. In these instances there are minor errors 
in the data presented by Olds, for which I have attempted corrections 
before calculating the coefficients. Olds does not give the data from 
which to calculate the coefficients that are significant at the .05 level. 
The method of calculating these coefficients as given in Table 1 is 
explained later. 

For N’s of 2 to 10, inclusive, Olds presents probabilities for each 
of the possible values of Sd?, excluding data in which tie rankings 
occur. To determine the exact point at which the coefficient is barely 
significant at each of the levels of significance, it was necessary to 
interpolate. Interpolation was done by means of the normal curve. 
The resulting coefficients, given in Table 1, separate clearly the dis- 
crete significant and non-significant coefficients which are possible 
when data involving tie rankings are excluded. Any type of interpo- 
lation would have accomplished this. The value of interpolation lies 
in providing a dividing line between significant and non-significant 
coefficients which because of tie rankings fall between the highest 
non-significant and lowest significant coefficients possible without tie 
rankings. In the absence of evidence concerning the distribution of 
coefficients possible in data involving tie rankings, a normal distribu- 
tion of them was assumed to be the best guess. Further consideration 
of this question is presented later. 


Adequacy of Olds’ Data for Testing Significance of Rank Difference 
Coefficients of Correlation 


All statements in this section apply only to coefficients obtained 
from data in which tie rankings do not occur. With this limitation it 
is fair to say that Olds’ (5) data provide the best basis yet available 
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for testing the significance of rank difference coefficients of correla- 
tion. 

The probabilities of various values of Sd? which Olds presents 
for N’s from two to seven, inclusive, are based upon tabulations of 
all possible combinations of ranks for each N. Kendall and others 
(4) have presented the results of similar tabulations which agree 
exactly with those of Olds. I have tabulated the possible combina- 
tions for N’s from two to six, inclusive, and these results also agree 
perfectly with Olds’ figures. Complete accuracy of Olds’ probabilities 
for N’s up to and including seven, therefore, may be assumed. 

For N’s of eight to ten, inclusive, Olds obtained approximate fre- 
quencies for various values of Sid? by computations based upon type 
II curves. Evidence is presented to show that for purposes of deter- 
ming levels of significance the approximations are very close (5). 
Kendall and others (4) present exact frequencies for N equals eight. 
I have compared the probabilities obtained from their frequencies 
with the probabilities calculated by Olds. There are slight differences, 
but the agreement is so close that in terms of dividing significant from 
non-significant coefficients for all of the levels listed in Table 1 of this 
report (still excluding from consideration all sets of data in which 
tie rankings occur) there are no disagreements. Olds’ data, there- 
fore, appear to be adequate for determining levels of significance 
for N’s of eight, nine, and ten. 

For N’s from 11 to 30, inclusive, Olds has calculated probabilities 
on the basis of assumed normal curves. Both Olds (5) and Kendall 
and others (4) have presented evidence showing that the larger the N 
the more reasonable is the assumption of normal distribution. Olds’ 
data indicate that while there are probably discrepancies between the 
exact distributions and his approximations for N’s above ten, these 
differences are in all probability small. While admittedly not com- 
pletely accurate, then, Olds’ probabilities for N’s above ten are the 
best available at present and for practical purposes are probably ade- 
quate in most instances. 

Inspection of Table 1 of this report reveals some discrepancies 
inherent in Olds’ data which are not apparent in Olds’ tables in which 
tabulations are in terms of Sd? only. The reader may spot these dis- 
crepancies by comparing the coefficients that are significant at the 
.01, .02 and .04 levels for N’s of 10 and 11. It is surely unreasonable 
to assume that in order to be significant at the .01 or .02 level a co- 
efficient found for an N of 11 should be higher than one found for an 
N of 10. Apparently the assumption of normal distribution has led 
to over-estimation of frequencies near the extremes of the distribu- 
tion for an N of 11. It is to be noted that the distortion decreases as 
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one moves in from the ends of the distribution and apparently be- 
comes a distortion in the opposite direction near the point outside of 
which five per cent of the cases fall at one end of the distribution 
(the point represented by .10 level of significance in Table 1). It 
seems fair to assume that both of these distortions decrease as N 
increases.* 


Comparison of Levels of Significance Obtained from Olds’ Data 
with Those Obtained by Other Methods 


The formula 
a 1.05 (1 — 7?) 


VN-1 ” 





has been suggested as a rough means of determining the reliability 
of a rank difference coefficient (2, p. 230). Now that we have em- 
pirical data or approximations to empirical data on the probabilities 
of various values of rho for N’s of 2 to 30, it is possible to test the 
adequacy of this formula as a basis for calculating significance. Two 
methods of testing the usefulness of the formula have been employed: 
(1) calculation by means of the formula of the ¢ ratios for coeffici- 
ents in Table 1 that are barely significant at the .01 and .05 levels; 
(2) calculation by means of the formula of the coefficients for se- 
lected N’s that would be barely significant at the .01 and .05 levels 
according to the ¢ ratio for the appropriate degrees of freedom 
(N-—2). The resulting data are given in Tables 2 and 3. Comparison 
of these data with the expected ¢ ratios and with the coefficients that 
are found to be significant by the empirical evidence from Olds indi- 
cates that the formula for the standard error of a rho is of little 
value in testing the significance of rho for N’s of 30 and below. The 
inappropriateness of the formula is greatest for small N’s and de- 
creases as N increases. The formula results in an over-estimation of 
the significance of rhos for all N’s up to 30. (Although the data for 
.05 level are given only for N’s up to 10, these statements apply to 
both the .01 and .05 levels for N’s up to 30. At the .05 level the over- 
estimation is slight for N of 30.) 

Since the above formula results in a too lenient test of signifi- 
cance, it was thought that perhaps the formula 


* Comparison of the coefficient (.816) for N equals 11 in the column for .01 
level of significance in Table 1 with the coefficient that one would obtain by extra- 
polation from the previous coefficients in the same column, indicates that the co- 
efficient .816 is erent not in error by more than .06. This error appears to be 
the largest that occurs anywhere in Table 1. 
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= 1.05 (3) 
saath var 4 


(assuming rho to be zero) would give better results. As the data in 
Tables 2 and 3 show, however, this formula results in a too conserva- 
tive estimate of the significance of rhos for N’s up to 30. As in the 
case of the first formula the inappropriateness is greatest for small 
N’s and decreases as N increases. 

Similar tests were made for the formula 


1 
=—— (4) 


¢, —>——_.. 
i va— i! 


The results are not presented here, but they indicate that this formula 
also yields too conservative an estimate of the significance of rhos for 
N’s up to 30. 

It appears that although one or more of the above formulas might 
be useful when N is large, none of them is satisfactory for determin- 
ing levels of significance of rhos obtained with N’s less than 30. 

One method of testing the significance of rhos at the .01 and .05 
levels has been found, however, which yields results surprisingly close 
to the empirical and approximated-empirical results from Olds’ data. 
This method consists of using the values given by Guilford [2, pp. 
323 f., adapted from Wallace and Snedecor (6), who in turn adapted 
their table from Fisher] for coefficients of correlation that are sig- 
nificant at .01 and .05 levels for various degrees of freedom and mul- 
tiplying each value by the constant 1.04. The close agreement between 
the figures obtained by this method and the figures obtained from 
Olds’ data may be seen in Tables 2 and 3. The only serious discrep- 
ancies for the .01 level of significance occur for an N of five and for 
N’s from 10 to about 15 or 18. In the case of these latter N’s we have 
already noted that there is reason to doubt the values obtained from 
Olds’ data. 

The discrepancies appear to be greater for the .05 level of sig- 
nificance, although still relatively small. Here we can compare the 
values obtained by taking 1.04 times Guilford’s figures with values 
from Olds’ data only for N’s up to 10, since Olds does not give the 
necessary data for N’s of 11 to 30. It is possible, however, to make 
rough interpolations from the data in Table 1 and obtain approxima- 
tions for the coefficients that would be significant at the .05 level for 
N’s of 11 to 30. It appears that the discrepancies at the .05 level be- 
tween the coefficients indicated by Olds’ data and the values obtained 
by taking 1.04 times Guilford’s figures increases as N increases from 
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TABLE 2 


Results Obtained by Various Methods of Determining the Rhos 
That are Barely Significant at the .01 Level 











N 2 3 4 5 6 7 
2 Gs) ec el . 9527 none 
3 none -990 none i 1.3487 none 
4 none -920 none * 1.6507 none 
5 none 850 none * 1.905+ .997 
6 .960 -795 none 25.946 2.043 .954 
7 .906 -752 none 11.766 2.112 909 
8 .869 -716 none 8.959 2.189 .867 
9 825 .687 none 6.933 2.224 .830 

10 788 653 none 5.925 2.251 -796 
11 .816 none 7.351 2.458 -764 
12 777 none 6.216 2.451 -736 
13 745 941 5.519 2.459 711 
14 -720 .889 5.148 2.474 .687 
15 .689 570 .846 4.687 2.452 .667 
16 666 807 4.411 2.458 648 
17 645 775 4,216 2.452 630 
18 626 T45 4.039 2.455 614 
19 .608 -716 3.897 2.462 .598 
20 .592 .512 .694 8.795 2.456 .583 
21 577 672 3.675 2.455 571 
22 563 652 3.609 2.459 558 
23 550 634 3.526 2.455 547 
24 .538 617 3.449 2.457 .536 
25 527 469 .601 3.400 2.463 .525 
26 .516 462 .587 3.351 2.457 .516 
27 .506 455 574 3.807 2.456 506 
28 497 449 .561 3.270 2.460 497 
29 488 442 .549 3.232 2.465 489 
30 479 486 539 3.193 2.456 482 








* t ratio is indeterminately high. 
+ Calculated by assuming rho to be 1.00. 


Column 1 lists the N’s. 

Column 2 lists the rhos calculated from Old’s empirical and approximated- 
empirical data (taken from Table 1 of this report). 

Columns 8 and 4 list the rhos that would be significant according to Guil- 
ford’s table of t ratios for N—2 degrees of freedom when the standard error of 








1.05(1 — r’2) 1.05 
rho is assumed to be —-——————— (column 3) or —————- (column 4). 
VN—1 VN—-1 


Columns 5 and 6 list the ¢ ratios obtained by dividing the values of rho ob- 
tained from Old’s data (given in column 2) by the standard error of rho found 
1.05(1 —7’2) 1.05 
by the formula ———————— (column 5) or ————— (column 6). 


Column 7 gives the values obtained for significant rhos by taking 1.04 times 
the values given by Guilford for r’s that are significant at the .01 level (in Table 
8, at the .05 level). 
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TABLE 3 


Results Obtained by Various Methods of Determining the Rhos that are Barely 
Significant at the .05 Level. (The descriptions of the 
columns are the same as in Table 2 above.) 











N 4 3 4 5 6 7 
2 re . ae 
3 none .948 none ae 1.3477 1.037 
4 none .827 none is 1.650+ .988 
5 .933 -745 none 13.721 LTT .913 
6 .843 .687 none 6.203 1.795 .843 
7 -781 .644 none 4.671 1.822 -784 
8 -725 .610 971 3.851 1.827 -735 
9 .681 581 .878 3.420 1.835 .693 

10 .644 .557 .807 3.145 1.840 .657 








* Indeterminately high. 
¢ Calculated by assuming rho to be 1.00. 


7 up to 30. Even at N equals 30, however, the discrepancy appears to 
be less than .02. In the absence of a better test of significance, even 
at the .05 level, the values obtained by multiplying 1.04 times Guil- 
ford’s figures might serve for N’s up to 30. These values apparently 
are slightly too conservative from N equals 7 up, and become increas- 
ingly so as N increases. Despite this weakness, which should be noted, 
the values obtained in this manner from Guilford’s figures have been 
listed in Table 1 as the least significant coefficients at the .05 level for 
N’s of 11 to 30. 

It was noted earlier that the figures in Table 1 for the .01 level 
of significance are out of line for N’s of 11 ff., because of the distor- 
tion resulting from the assumption of normal distributions. The for- 
tunate circumstance that the values obtained by 1.04 times Guil- 
ford’s figures fit so closely the values in Table 1 for N’s up to 10 and 
for N’s of 26 to 30 suggests a simple method of obtaining an approxi- 
mate correction for the distortion. This consists of substituting for 
the figures in the .01 column of Table 1 for N’s of 11 to 25, inclusive, 
the figures in Table 2 obtained by taking 1.04 times Guilford’s values 
for coefficients of correlation that are barely significant at the .01 
level for the corresponding N’s. This amounts to making rough inter- 
polations of the values for N’s of 11 to 25 from the values for N’s up 
to 10 and from N’s of 26 to 30. The values obtained by means of this 
rough interpolation are listed in parentheses in Table 1. 

Since values equal to 1.04 times Guilford’s values agree so closely 
with the coefficients obtained from Olds’ data for the .01 level of sig- 
nificance for N’s of 25 to 30, it would appear reasonable to assume 
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that for N’s immediately above 30 one can obtain a close approxima- 
tion to the coefficients that are significant at the .01 level by taking 
1.04 times Guilford’s figures (2, pp. 323 f., after Wallace and Snede- 
cor, who interpolated from Fisher’s data) .* 


The Problem of Tie Rankings and Tests of Significance 


Neither Olds (5) nor Kendall and others (4) consider the prob- 
lem of tie rankings in relation to tests of significance for coefficients 
of rank difference correlation. Perhaps the problem is unimportant, 
and yet a casual survey of some of the combinations possible when tie 
rankings occur with an N of four suggests that when N is very small 
one or more pairs of tie rankings will change very greatly the fre- 
quencies with which various values of Sd? and rho can be obtained. 
When N equals four, for example, there are at least seven ways of 
getting a Sd? of zero (or arho of + 1.00) when tie rankings occur. In 
addition there are at least seven ways of getting Sd? greater than 
zero but less than two. By contrast, out of the 24 possible combina- 
tions obtainable with an N of four in data in which no tie rankings 
occur, there is only one way of obtaining a Sd* of zero and three 
ways of obtaining Sd? of two. 

It seems probable that as N increases, and surely as the ratio of 
tie rankings to untied rankings decreases, the effect of tie rankings 
upon the probabilities of obtaining various values of Sd? and rho will 
decrease in importance. It would be futile, however, to tabulate or 
calculate all the possible combinations that could occur for various 
N’s in data involving tie rankings; for there is no possible way of de- 
termining adequately the probability of the ties themselves occur- 
ring.** The probability of ties occurring will vary with the type of 


* Inspection of Tables 2 and 3 suggests the hypothesis that as N increases 
the formula 1.04 times Guilford’s figures yields increasingly too conservative esti- 
mates of the coefficients that are significant at both the .01 and .05 levels. This 
tendency appears to begin at N equals 7 for the .05 level. Where it begins for 
the .01 level of significance is not clear from these data; perhaps it is at N equals 
7, N equals 9, or N equals 29. It is possible that for large N’s (probably larger 
for the .01 level than for the .05 level of significance) Guilford’s figures for sig- 
nificant coefficients of correlation would apply directly without correction. The 
evidence for this suggestion is admittedly weak, although rough interpolation in 
Table 1 for the value for N of 30 at the .05 level of significance indicates that 
the value obtained would be approximately that given in Guilford’s table. 

** Tt would be possible to tabulate separately all of the possible combinations 
of pairs of ranks for various N’s, first for data involving a single instance of a 
tie ranking involving two cases and occurring in only one of the twe sets of 
ranks, second for data involving a single instance of a tie involving two cases in 
both sets of ranks, third for data involving a single set of ties between three cases 
in one set of ranks, and so on for all possible types, numbers, and combinations 
of ties. Tables of probabilities might then be set up separately for each of the 
numerous types of data involving ties. The task would be well nigh endless and 
seems scarcely worth while or practicable. 
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data involved and with the methods used in collecting and treating 
the data—whether tie judgments are permitted, to how many decimal 
places calculations are carried, etc. 

Another problem arises in data involving tie rankings. For the 
case where N is four, there are at least seven ways of obtaining a 
rho of +1.00, but there is no way of obtaining a rho of —1.00 so long 
as ties are present in either set of ranks and the usual formula for 
rho is used. The question arises as to whether tie rankings tend to 
increase the probability of positive coefficients and to decrease the 
probability of negative coefficients. If so, it then becomes pertinent 
to inquire to what extent the probabilities of positive and negative 
coefficients are changed by the presence of ties. Perhaps it will be 
found necessary to introduce a change in the denominator of the for- 
mula for rho for data involving tie rankings. 

It appears probable that the importance of the questions raised 
in the preceding paragraph will decrease as N increases and as the 
ratio of tie rankings to untied rankings decreases. Nonetheless, these 
questions along with the, for this paper, more pertinent general prob- 
lems stated above should serve to create doubt concerning the adequacy 
of any method of testing the significance of rhos obtained from data 
in which tie rankings occur. In the absence of evidence concerning 
the probabilities of various values of rho for data involving ties, the 
levels of significance given in Table 1 probably represent the best 
means available at present for testing the significance of such rhos. 
When N is relatively large and the number of ties is relatively small, 
the problem of the effect of ties upon the significance of a coefficient 
can probably safely be ignored. When N is small and when the num- 
ber of ties is relatively large, it perhaps should be obligatory upon an 
experimenter to report the number and nature of the tie rankings as 
well as the rho and the N. Certainly in this latter case any statement 
concerning the level of significance of a rho is open to question. 


Recommendations 


From the data presented in this paper a few recommendations 
concerning tests of significance of rank difference coefficients of corre- 
lation seem justified: 

(1) When calculations are made by the usual procedure using 
Spearman’s formula, Old’s (5, pp. 145-148) tables of probabilities for 
various values of Sd? for N’s from 2 to 30 are the most convenient 
and adequate means of testing significance. The few minor errors in 
Olds’ tables, noted earlier in this paper, should be corrected, however; 
and the distortions present in the probabilities for N of 11 and N’s 
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immediately above 11 should be noted. 

(2) When testing the significance of rhos reported in the litera- 
ture or calculated by methods not involving the finding of Sd’, the 
values given in Table 1 of this report will be useful for N’s from 2 to 
to 30. When testing significance at the .01 level, where alternate val- 
ues are given in Table 1, the values given in parentheses are prefer- 
able since they have been corrected for the distortion present in Olds’ 
probabilities for N’s of 11 ff. In testing significance at other levels 
than .01 for N’s of 11 and above, the levels of .04 and .10 are prefer- 
able because the values appear to be more accurate for these levels 
of significance than for the .02 and .05 levels. For N’s of 10 and be- 
low the values in Table 1 may be considered quite exact—completely 
so for N’s of 8 and below for coefficients obtained from data not in- 
volving tie rankings. 

(3) A close approximation to the rank difference coefficients 
that are barely significant at the .01 level for N’s immediately above 
30 is probably obtained by taking 1.04 times the values given for N—2 
degrees of freedom in Guilford’s adaptation (2, pp. 323 f.) of Wal- 
lace and Snedecor’s table of coefficients of correlation that are sig- 
nificant at the .01 level. Perhaps the values given in the same table 
for coefficients significant at the .05 level may be taken directly as 
the rhos that are significant at the .05 level for N’s immediately above 
30; this latter is less well justified by the evidence, however. 


1.05 (1 — 7) 1.05 1.00 


———_, ———, an for 
Mae | VN -1 VN-1 








(4) The formulas 


calculating a standard error of a rho appear inaedquate for purposes 
of calculating t ratios by which to test the significance of a rank dif- 
ference coefficient of correlation for N’s up to 30. The first formula 
results in too high ¢ ratios; the other two formulas result in ¢ ratios 
that are too low. 


(5) When tie rankings occur in the data from which a rho is 
calculated, caution should be used in evaluating the level of signifi- 
cance of the coefficient. The data given in Table 1 of this report prob- 
ably represent the best means available at present for testing the sig- 
nificance of such a coefficient; but whenever the number of tie rank- 
ings is relatively large in comparison to the N of the ranks, and es- 
pecially when N is very small, any test of significance should probably 
be considered questionable. 
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An outline for a course in test theory is presented, together 
with a list of assignments, problems, and a bibliography. The course 
has been given in the Psychology Department of the University of 
Chicago. The material is presented in outline form at the present 
time because of the increased need for training in test theory due 
to the increase in the use of psychological tests for classification of 
military personnel, and because much of the material in such a 
course must be selected from a wide array of articles in the litera- 
ture. This material is presented in order that an organized body of 
material for instructional purposes may be readily available to 
those interested. 


The recent increase in psychological testing for the purpose of 
classifying military personnel has increased the need for training in 
the quantitative theory which is basic to the construction of tests and 
the analysis of test data. 

During the last forty years, progress has been made toward a 
well-integrated, quantitative theory pertaining to the behavior of test 
items and test scores under different conditions. This rationale of 
mental tests should not be confused with “statistics.” A good foun- 
dation in elementary statistics is a prerequisite for a course in the 
theory of mental tests. 

At present the work on the theory of mental tests is for the most 
part not available in text-book form. It is necessary to select the 
reading material for such a course from a wide array of journal ar- 
ticles. The writer has prepared an outline, bibliography, and prob- 
lems for such a course. This material is presented here in order that 
an organized body of material on test theory for instructional pur- 
poses may be readily available to those interested. 

This course deals with the mathematical and statistical theory 
necessary for interpreting test results. It does not deal with the non- 
quantitative problems involved in the construction of aptitude or 
achievement tests; nor does it attempt to familiarize the student with 
the various psychological and educational tests now available. How- 


* On leave from the University of Chicago for a government research project 
at the College Entrance Examination Board, Princeton, New Jersey. 
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ever, reference is made to articles and books dealing with material 
of this nature, and the student is urged to familiarize himself with 
this material in case he has not already had work dealing with these 
topics. 

The material on test theory is presented in four sections. First 
an outline of the course, then a specific list of assignments, followed 
by a set of problems and the bibliography. In the preparation of the 
bibliography, a good foundation in elementary statistics, including 
correlation, was assumed. For some of the articles a knowledge of 
multiple correlation and matrix theory is necessary. It was found de- 
sirable in preparing the material for class use to have the assignments 
as compactly presented as possible, so that the student could easily see 
just what had to be done. The assignments follow the same numbering 
system as the outline, so that it is possible to refer readily to the 
assignment for any part of the outline, or to look up that part of the 
outline corresponding to any assignment. 


OUTLINE 


I. Introduction to the course and review of statistics. 
(six class periods, with review examinations on statistics in 
the third and fifth periods. The examinations are returned 
and discussed during the fourth and sixth sessions.) 


II. Accuracy of test scores. 
A. Different measures of accuracy of test scores. 
1. Standard error of measurement. 
a. Estimation of test error usually made by using two 
comparable tests on the same population. 
b. Assumptions usually used to define test error—the 
basic assumptions of test theory. 
(chance error distinguished from constant and 
systematic errors) 


(1) The obtained score s of any person may be di- 
vided into two parts; a true score t which rep- 
resents his actual ability, and an error score e 
which is due to the various factors that may 
cause a person to answer correctly an item 
which he does not know, or to answer incor- 
rectly an item which he does know. Specifical- 
ly it is assumed that these three scores, s, ¢ and 
e are related as follows: 

s=t+e, or e=s-t. 
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(2) The sum of chance error scores for the group, 
and hence the average error, is zero. This may 


be written 
dSe=0, or M.=0. 
(3) The correlation of error scores with true scores 
is zero. This may be written 
Set=0, or Te =O. 


(4) The correlation of error score for one test with 


error score for another test is zero. A special 
case of this assumption is that the correlation 
between error scores on two parallel forms of 
the same test is zero. This may be written 


2e.e,—90, or ,e,—9. 


(5) The standard deviation of the distribution of 
error scores on one form of a test is equal to 
the standard deviation of the distribution of 
error scores on a parallel form of that test. 
This may be written 

yYee= Te, or oa 


ir 


1 2 


(6) The true score on one test for a given person 
is equal to the true score of that person on a 
parallel form of the test. That is 

t=, 
[From assumption (6) it follows that 
o:,=or,, M:,=M:,, and 
T:.,—1.] 


Estimation of parameters of distribution of true 
scores, and of error scores. 


Definition of equivalent forms of a test. 


(1) Items in form 1 psychologically similar to 
items in form 2. 


(2) M,=M.. 
(3) 071 = 02 ° 


Test variance equals sum of true variance and er- 
ror variance. 


Interpretation of standard error of measurement. 
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(1) Standard deviation of distribution of error 
scores. 

(2) Error made in using obtained scores instead of 
true scores. 

(3) Error made in predicting obtained scores from 
true scores. 

(4) Obtained variance minus true variance. 


Reliability coefficient and index of reliability. 

a. Standard error of measurement expressed as a 
function of test variance and reliability. 

b. Reliability coefficient as ratio of true to total vari- 


ance. 
c. Interpretation of coefficient and index of reliabil- 


ity. 
Standard error of estimate. 


a. Of test score. 
b. Of true score. 


Standard error of substitution. 
a. Derivation. 
b. Interpretation. 


Coefficient of alienation. 


Comparison of various measures of accuracy of test 
scores. 


B. Factors affecting accuracy of test scores. 


1. Effect of increased. dispersion on reliability and stand- 


ard error of measurement. 
a. Increased dispersion due to increased range of abil- 
ity. 


b. Increased dispersion due to increased test error. 


2. Length of test. 


a. Effect of doubling test length. 

b. Effect of further increasing length of test. 

c. Prediction of length necessary for any given re- 
liability. 

d. Experimental verification of c. 


8. Difficulty of test. 
Inter-item r’s. 











C. 
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Methods of estimating test reliability. 


Repetition of same form. 

Equivalent forms. 

Split-halves method. 

Reliability of time-limit tests. 

Method for unique estimation of reliability from para- 
meters of test score distribution. 

Estimating reliability from mean, variance, and num- 
ber of items. 

7. Reliability coefficient calculated from variance of sums 
and variance of differences of halves. 


ee ee 


- 


Special topics in reliability. 
1. Reliability of a sum of variables. 
r of any sum. Various special cases of this. 
2. Reader reliability of a test. 
3. Content reliability of a test. 
4. Common factor interpretation of reliability. 


Validity of a test. 

1. Definition of validity (correlation of test with criter- 
ion). 

Relation to the concept of reliability. 

Relationship between length and validity. 

Length required for given validity. 

Relationship between reliability and validity. 
Correction for attenuation. 

Relation to difficulty. 


SS Om wp 


III. Scoring Methods. 


A. 


Linear transformations. 

1. Deviation score. 

2. Standard scores. 

3. Derived scores with any given mean and standard dev- 
iation. 

Non-linear transformations. 


1. Percentile. 

2. Normalized scores. 

3. McCall’s T-score. 

4. Scaled scores of the Cooperative Test Service. 


Age scales and other scales depending upon an external 
criterion. 
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Mental age. 
Intelligence quotient. 
3. Criticism of the mental age concept. 


D. Absolute scaling. 
Factors affecting the shape of the test score distribu- 
tion. 


3 


2. 


a. 


b. 
- 
d 


Number of items. 

Inter-item correlation. 

Item difficulty distribution. 

Test score is not necessarily linear with any true 
measure of ability. 


Problem of a constant unit of measurement (difficulty 


and the discrimination function). 


38. An absolute zero point. 
4. Applications of absolute scaling methods. 


E. Scoring formulas. 


go PO 


4, 


Correction for chance successes. 
Empirical scoring formulas using multiple correlation. 
Time limit vs. amount limit tests. 


a. 
b. 


Power vs. speed tests. 
Variance of distribution of number omitted. 


Scoring of rank-order items. 


IV. Combination of Measures. 


A. Combination of test scores. 
To maximize validity, given an external criterion. 


i, 


2. 


a. 


b. 


Multiple correlation. 

Caution on multiple correlation. 

Approximations to multiple correlation (see IV, B, 
1, b). 

Group difference and other methods (see IV, B, 1, 
c). 


To maximize reliability or internal consistency when 


no external criterion is available. 


Inter-test correlations. 

Maximizing inter-individual variance. 

Minimizing inter-test variance. 

The one-factor approach. 

Minimizing generalized variance of all individuals 
receiving the same score. 
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b. Use of approximations to inter-test correlations 
(see IV, B, 2, b). 
ec. Other methods. 
By dispersions. 
By reliabilities. 
By difficulty. 


3. Irrelevance of weights when number of variables is 
large or when inter-correlations are high. 


4. Comparison and criticism of methods (see IV, B, 4). 


B. Selection of test items. 


1. To maximize validity (given an external criterion). 


a. Multiple correlation (see IV, A, 1, a) 
(usually too laborious when dealing with items). 
b. Approximations to multiple correlation. 
L-method. 
Successive residuals. 
Maximizing function. 
Rapid approximation to L-method. 
Flanagan’s method. 
c. Group difference and other methods. 
Weighting of items in a 
personality test. 
interest test. 
application blank. 
(weighting in a test where best ordering of test 
responses cannot be prejudged). 
Fisher, analysis of variance. 
With three or more possible answers weight to 
maximize correlation. 
Theoretically the multiple correlation approach is 
best, but in many cases it is too laborious. Other 
methods may be regarded as an approximation to 
this method, either to save work, or because the 
criterion is missing. 
2. To maximize reliability or internal consistency when 
no external criterion is available. 
a. Use of inter-item correlations (see IV, A, 2, a). 
Usually too laborious when dealing with items. 
b. Use of approximations to inter-item correlations. 
Use of item-test correlation. 
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c. Other methods (see IV, A, 2, c). 

8. Irrelevance of weights when number of variables is 
large, or when inter-correlations are high (see IV, 
A, 8). 

4. Comparison and criticism of methods. 
Guilford. 
Adkins. 
Long. 
Lentz. 
Merrill (homogeneity). 
Mosier (one-factor approach). 


V. Relationship between test theory and psychophysics. 


Assignments 
General directions for assignments 


Textbook and references. 

The principal text is Thurstone, 1931, The Reliability and Valid- 
ity of Tests. This book is referred to as Thurstone 1931. J. P. Guil- 
ford’s Psychometric Methods is recommended for those who wish to 
use two texts. The references in the assignments are given by author 
and date. The complete reference may be found in the bibliography, 
listed alphabetically by author’s name. 


Problems. 


The problems accompanying the assignments are either from 
Thurstone’s text, or from the list of supplementary problems given 
after the assignments. The former are indicated by numbers, the 
latter by letters. 


Suggestions for preparing the problems 

1. The following materials will be needed: ruler, compass, graph paper, data 
sheet paper, (11” x 17”). 

2. Leave a 1 1/2 inch margin so that the papers can be bound without conceal- 
ing the work. 

8. Make all graphs on printed graph paper. 

4. The best way to bind the data sheets is to fold them with the ruled surface 
out and put both edges into the binding. This necessitates leaving a 1 1/2-inch 
margin on both the right and left edges and not using the column which 
comes at the fold. 


5. In making graphs use an x for each point. This method makes it easy to see 
the general trend of the graph and yet secures accuracy since the intersec- 
tion of the two lines of the cross is placed at the exact point to be indicated. 
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6. Make all computations neatly on the first copy so that the original data sheet 
can be handed in. Never do your figuring on scratch paper and then copy it 
to hand in. This method both takes additional time and increases the chances 


of error. 


7. Label each problem, giving the book, chapter, page, and problem number. 
8. The problems for each assignment should be bound in a folder and handed in 
as a unit with your name and the assignment indicated. 


Assignment I 


Readings: 


Assignment II-A 
Readings: 


Topic 2 
Readings: 


Problems: 


Topic 3 
Readings: 


Problems: 


Topic 4 
Readings: 


Topic 5 
Readings: 


Problems: 


Topic 6 


Thurstone 1924 Ch. 1-3, 6, 7, 10-15, 17-19; Ch. 4, 8, 9, 21-25 


Thurstone 1931 preface, pp. 1-4, 17-20 
Boring, 1920 


Thurstone 1931 pp. 12-16, 23 
Guilford 1986 (b) pp. 408-421 
Thurstone 1981 (8, 9, 10, 11, 12, 18, 25, 26) 


Thurstone 1931 pp. 52-56 
Douglass 1934 

Monroe 1934 

Exercise B 


Kelley 1927 pp. 211-213, 171-181 


Thurstone 1931 pp. 57-61 
Thurstone 1931 (2, 16) 


No special readings or problems 


Assignment II-B 
Topic 1 
Readings: 
Problems: 


Topic 2 
Readings: 


Problems: 


Topic 3 


Thurstone 1981 pp. 24-27 
Otis 1922 (b) 
Thurstone 1931 (28, 29, 30, 31) 


Thurstone 1931 pp. 28-36, 40-45 

Holzinger and Clayton 1925 

Ruch et. al. 1926 

Thurstone 1981 (5, 6, 7, 18, 17, 20, 22, 34, 35, 36) 


No special readings or problems 


Topic 4 
Readings: 


Richardson 1986 (a) 
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Assignment II-C 
Topic 1-4 


Readings: 


Topics 5-6 


Readings: 


Problems: 


Topic 7 


Readings: 


Problems: 


Assignment II-D 
Topic 1 


Readings: 


Topic 2 


Readings: 


Topic 3 


Readings: 
Problems: 


Topic 4 


Readings: 


Assignment II-E 
Topics 1-2 


Readings: 


Problems: 


Topics 3-4 


Readings: 
Problems: 


Topic 5 


Readings: 
Problems: 


Topic 6 


Readings: 


Problems: 


Topic 7 


Readings: 
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Thurstone 1931 pp. 5-11 


Kuder and Richardson 1937 
Dressel 1940 

Richardson and Kuder 1939 
Exercise K 


Rulon. 1939 
Otis and Knollin 1921 
Exercise H 


Kelley 1924 pp. 196-200 
Walker 1929 pp. 112-118 


Starch and Elliot 1912, 1918 (a), 1918 (b) 
Monroe 1938 


Gulliksen 1936 
Exercise J 


Burt, Sections I-V, emphasizing pp. 280-297 


Thurstone 1931 pp. 97-104 
Otis 1922 (a) 

Guilford 1936 (b) pp. 421-426 
Segel 1933 

Thurstone 1981 (8, 4, 24) 


Thurstone 1931 pp. 46-49 
Thurstone 1981 (21, 37, 38, 39) 


Thurstone 1931 pp 50-51 
Thurstone 1931 (14, 38, 40, 41) 


Thurstone 1931 pp. 62-68 

Walker 1929 pp. 118-123 

Spearman 1907, 1910 

Brown and Thomson 1921 Ch. 8 
Thurstone 19381 (1, 15, 28, 42, 48, 47) 
Exercise A 


Richardson 1936 (b) 
Thurstone, T. G. 1932 




















Assignment III-A 


HAROLD GULLIKSEN 233 


Topics 1-3 incl. 


Readings 
Problems 


Assignment III-B 
Topics 1-2 

Readings 

Problems 


Topics 3 and 
Readings 


Assignment III-C 
Topics 1-3 


Readings: 


Assignment III-D 
Topics 1-2 


Readings: 


Problems: 


Topic 3 


Readings: 


Topic 4 


Readings: 


Assignment III-E 
Topics 1-4 


Readings: 


Problems: 


Assignment IV-A 
Topics 1 


Readings: 


Problems: 


Topic 2 (a) 


Readings: 


: Thurstone 1924 Ch. 15 
: Exercise C 


: Thurstone 1924 Chs. 16 and 20 
: Exercise D 


4 
: McCall 1922 Ch. 10 
Flanagan 1939 


Freeman 1917 

Freeman 1939 Ch. IV 
Thurstone 1926 

Toops and Symonds 1922 
Yerkes, et. al. 1915 
Stern 1914 p. 80 


Kelley 1923 
Thurstone 1925 
Exercise E 


Thurstone 1928 


Flanagan 1939 
Thorndike 1922 


Thurstone 1931 pp. 69-82 
Thurstone 1919 

Moore 1940 

Guilford 1986 (a) 
Thurstone 1981 (19) 
Exercises F and I 


Garrett 1943 

Guilford 1936 (b) pp. 380-399 
Frisch 1934 

Exercise G 


Burt 1986 (Section VI, pp. 297-304) 
Edgerton and Kolbe 1936 

Horst 1986 (a) 

Wilks 1938 
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Topic 2 (c) 


Readings: 


Problems: 


Topic 3 


Readings: 


Assignment IV-B 
Topic 1 (b) 


Readings: 


Topic 1 (c) 


Readings: 


Problems: 


Topic 2 (b) 


Readings: 


Topic 4 


Readings: 


Assignment V 
Readings: 
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Thurstone 1931 pp. 83-90 
Richardson: 1936 (b) 
Thurstone, T. G. 19382 
Thurstone 1981 (44, 45) 


Stalnaker 1938 

Stalnaker and Richardson 1933 

Wilks 1938 

Burt 1936 

Lee and Symonds 1934 (for review of studies on weighting) 


Adkins and Toops 1937 


Flanagan 1936 
Horst 19384 (a); 1934 (b); 1936 (b) 


Thurstone 1931 pp. 91-96 
Boring 1919 

Lindquist 1940 

Travers 1939 

Thurstone 1931 (32, 46, 48, 49) 


Babitz and Keys 1940 
Edgerton and Toops 1928 
Richardson 1936 (a) 


Guilford 1936 (b) pp. 426-437 
Adkins (Thesis) 

Lentz, et al 1932 

Long, et al 1935 

Merrill 1937 

Mosier 1936 

Swineford 1936 


Guilford 1937 
Mosier 1940 


For bibliographies on tests and testing see 
Buros 1936, 1937, 1939, 1941 
Hildreth 1939 
Holmes 1917 
Lee and Symonds 1934 
National Society for Study of Education 1918 


Ruger 1918 


Whipple 1910 


For methods of constructing tests and test items see 
Board of Examinations Chicago 1937 
y Englehart 1942 
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Hawkes, et al 1986 
Hull 1928 
Orleans 1987 


Notes on arithmetic and graphic computing methods 
Dunlap and Kurtz 1932 
Dunlap 1936 (a) and (b) 
Kuder 1987 
Richardson 1935 


General discussion of tests 
Freeman 1939 
Kelley 1927 
McCall 1922 
Monroe 1923 
Monroe and Englehart 1936 
Orleans 1987 
Ruch and Stoddard 1927 
Smith 1938 
Stern 1914 


PROBLEMS 


PROBLEM A 
Derive the equation for predicting scores in X, from scores in X, where 
X, and X, are two different forms of a test whose reliability and validity are 
both known. Derive the equation for predicting the true score Y from Y, , where 
Y, is a test whose reliability and validity are known. 


PROBLEM B 
Describe the correct experimental method for checking the applicability of 
the formula 


= _ 2 
o,—9%,V1 — 7, 


where o, is the standard error of estimate made in 
estimating y from x; 


o, is the standard deviation of the observed 
distribution; and 
1, is the correlation between x and y. 


PROBLEM C 
Compute the table and draw the graph which would be used for transform- 
ing raw scores in the A. C. E. (1937 edition) (see information in accompanying 
table) into: 
a, deviation scores (d-scores) 
b. standard scores (z-scores) 
c. derived scores with a mean of 50 and a standard 


deviation of 10 (D-scores) 
Consider only the total distribution given in the right-hand column of the table. 
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TABLE FOR USE IN CONNECTION WITH PROBLEMS C AND D 

The following scores were made by 68,899 students in 323 colleges on the 
1937 edition of the American Council on Education Psychological Examination 
for College Freshmen: 


Frequency 

Scores Men Women Total* 
0-9 2 4 
10-19 5 3 12 
20-29 27 22 58 
80-39 85 50 170 
40-49 169 112 329 
50-59 829 225 626 
60-69 471 358 943 
70-79 667 479 1,314 
80-89 923 769 1,915 
90-99 1,108 892 2,264 
100-109 1,387 1,171 2,897 
110-119 1,669 1,376 3,429 
120-129 1,768 1,529 3,764 
130-139 2,064 1,768 4,348 
140-149 2,113 1,793 4,471 
150-159 2,188 1,830 4,650 
160-169 2,220 1,748 4,600 
170-179 2,128 1,798 4,583 
180-189 1,990 1,610 4,207 
190-199 1,823 1,479 3,904 
200-209 1,639 1,351 8,593 
210-219 1,488 1,251 3,281 
220-229 1,234 996 2,686 
230-239 1,097 906 2,441 
240-249 893 748 2,025 
250-259 750 596 1,630 
260-269 584 488 1,309 
270-279 474 829 998 
280-289 358 273 772 
290-299 284 187 580 
800-309 184 122 387 
310-319 153 74 286 
320-329 96 52 181 
330-339 70 38 133 
340-349 29 13 51 
350-359 24 9 40 
360-369 6 3 14 
870-379 2 2 
380-389 1 2 
Total 32,500 26,450 68,899 
Lower quartile 127.27 127.54 128.67 
Median 165.75 164.84 167.08 
Upper quartile 207.57 206.10 208.87 


* The total includes the scores of 9,949 students not classified according to sex. Data taken 
from Thurstone, L. L. and Thurstone, T. G. The 1937 Psychological Examination for College Fresh- 
men. The Educational Record, April, 1938, pp. 209-234. 


PROBLEM D 


Compute the table and give the graph which would be used to change raw 
scores on the A. C. E. (1937 edition) into 


a. percentile scores (p-scores) 
b. normalized scores (N-scores) 
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Again use only the total distribution given in the right-hand column. 
Referring also to problem C draw the graphs showing the relation between 


a. D-scores and z-scores 
b. D-scores and p-scores 
. D-scores and N-scores 
. 2-scores and p-scores 
. z-scores and N-scores 
. p-scores and N-scores 
. Plot p-scores against D-scores on arithmetic prob- 
ability paper as a test of normality 
Write a brief paragraph stating the relationship shown in the foregoing six 


graphs. 


i. Me ae) 


PROBLEM E 


Below is given the frequency distribution of A. C. E. scores for 113 students 
taking the 56 tests used in Mr. Thurstone’s first large study of primary mental 
abilities. (Thurstone, L. L. Primary mental abilities, p. 19). This distribution 
is given in terms of percentile points on the national norms for the A. C. E. test. 
The table shows that there was one student between the 35 and 40 percentile 
points on the national norms; two students between the 45 and 50 percentile 
points; and so forth. It will be noticed that over 25 per cent of the students are 
above the 98 percentile point on the naticnal norms. Can this distribution of 113 
cases be regarded as a normal distribution, granted the assumption of a Gaussian 
distribution of intelligence in the forty thousand students on whom the national 
norms were based? Use the absolute scaling methods to answer this, question. 


cumulative 
scores* frequency frequency 
35-40 i 1 
40-45 + 2 
45-50 2 4 
50-55 2 6 
55-60 4 10 
60-65 6 16 
65-70 2 18 
70-75 4 22 
75-80 10 32 
80-85 6 38 
85-90 10 48 
90-95 25 73 
95-96 3 76 
96-97 4 80 
97-98 3 83 
98-99 6 89 
99-999 23 112 
999-1.000 1 113 


* National norms (1933), 40,229 cases, 208 colleges. 


PROBLEM F 
Derive the formula for the correlation between number correct and number 
incorrect for an objective test, assuming that there are no omissions. 
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PROBLEM G 
Entering freshmen at the University of Chicago are given an A .C. E. Psy- 
chological Examination (a), a physical sciences aptitude test (s), an English 
placement test (e). A year later they are given the physical science compre- 
hensive (p), and the humanities comprehensive (h). 
The following zero-order correlations are obtained: 
7,.== 40, ¢.,.==.R, %%.— 4, 
Tap — 50, T,,=.70, 1ep—.40, 
T 1, — 60, 1,,—=.20, 7,,—=.70, Ty, — 60. 
The following means and standard deviations are found: 
a 8 e p h 
Mean 120 110 150 220 460 
Standard deviation 30 20 25 30 40 


1. Write the equation for making the best prediction of the humanities com- 
prehensive score from the three placement tests. 

2. What will be the correlation between the predicted humanities scores and the 
actual scores, using the prediction formula given in 1? 

8. Which two placement tests will give the best prediction of scores in the physi- 
cal-science comprehensive? 

4. Write the equation for making the best prediction of the physical-science com- 
prehensive score from the two tests mentioned in 3. 

5. What is the correlation between the actual physical-science scores and the 
scores predicted by using the equation given in 4? 


PROBLEM H 

Make the following calculations from the data given in the appendix. 

Estimate the reliability coefficient for the total test by the split-half method 
using the Spearman-Brown correction. 

Estimate the reliability coefficient from the variance of the total score and 
the variance of the difference between scores on the two halves. 

Give the standard error of measurement for this test. 

What are the probable limits for the true score of a person who scores 51; 
one who scores 73. 

Estimate the reliability of a test which is twice as long as this one, four 
times as long, seven times as long, ten times as long (see table for Spearman- 
Brown formula in Dunlap and Kurtz). Estimate the reliability the test would 
have if it were made infinitely long. If one wished this test to have a reliability 
of .97, how long would it be necessary to make it? Graph the foregoing results. 

Estimate the reliability this test would have if it were applied to a group 
whose scores had a standard deviation half that of the original group. Estimate 
the reliability this test would have if it were applied to a group whose scores had 
a standard deviation twice that of the original group. 


PROBLEM I 
Comment briefly on the material in Moore’s 1940 article. 


PROBLEM J 


An examination has an objective section (0) and an essay section (e). The 
split-half correlation of scores for section o is .80. The corresponding correlation 
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for section e is .65. On section e the correlation between the total score given 
by reader A and reader B is .85. 
Estimate the content reliability for section 0; for section e. 
PROBLEM K 
From the data given in the appendix, estimate the reliability of the test by 
the Kuder and Richardson method. Use equations 8, 14, 20, and 21. (Kuder and 
Richardson 1937). 
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APPENDIX 


The following data on 52 students taking the composition section of the French 104-5-6 in 
of Examinations at the 


June, 1940 were made available by Dr. Lawrence Andrus of the 


University of Chicago. 
Column A gives the item number. 


Column B gives the proportion passing above mean. 
Column C gives the proportion passing below mean. 


Column D gives the proportion of entire group passing. 


Column E gives 7 (tetrachoric). 











> 
=e] 
Q 
o 


1 21 04 13 
2 68 38 54 
3 57 25 42 
4 68 42 56 
5 82 63 73 
6 32 21 27 
7 71 58 65 
8 1.00 1.00 1.00 
9 39 .08 25 
10 29 13 21 
11 64 42 54 
12 57 a bf 38 
13 75 42 60 
14 68 50 60 
15 93 58 ad 
16 Bb 42 62 
17 64 54 60 
18 75 46 62 
19 96 -79 88 
20 57 25 42 
21 86 79 83 
22 64 58 62 
23 75 42 60 
24 86 67 77 
25 xf 54 63 
26 64 a Wy 42 
27 19 29 56 
28 86 -79 83 
29 57 42 50 
30 86 71 19 
31 my fi 67 69 
82 36 17 27 
33 68 33 52 
34 68 54 62 
35 75 75 75 


37 64 88 75 
38 43 46 44 
64 
40 1.00 1.00 1.00 
41 96 1.00 98 
42 71 71 71 
43 71 63 67 
44 82 42 63 
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I II cs Gime i's I i. tt 
1 41 24 17 27 58 32 26 
2 40 22 18 28 35 24 11 
3 73 40 33 29 55 31 24 
4 39 20 19 30 62 82 30 
5 74 37 37 31 68 32 36 
6 49 31 18 82 55 30 25 
7 35 20 15 33 62 29 33 
8 59 33 26 34 67 36 31 
9 44 28 16 35 53 30 23 
10 51 25 26 36 54 29 25 
11 55 26 29 37 61 32 29 
12 54 31 23 38 68 31 37 
13 36 25 11 39 58 30 28 
14 74 35 39 40 60 29 31 
15 48 29 19 41 84 43 41 
16 52 28 24 42 39 20 19 
17 66 42 24 43 37 22 15 
18 73 39 34 44 56 28 28 
19 59 33 26 45 56 31 25 
20 50 26 24 46 25 15 10 
21 25 18 ‘ij 47 72 36 36 
22 60 31 29 48 33 24 9 
23 60 34 26 49 41 26 15 
24 65 34 31 50 66 35 31 
25 41 18 23 51 38 27 11 
26 65 35 30 52 84 Al 43 


Column I gives the code number of each student. 

Column II gives the total score for each student. 

Column III gives the score on the first fifty items for each student. 
Column IV gives the score on the second fifty items for each student. 


Number of items = 100. 

Maximum number of score points = 100. 
Mean raw score — 54.52. 

Standard deviation = 13.98 

Number of students = 52. 
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THE CONCEPT OF TEST AND ITEM RELIABILITY 
IN RELATION TO FACTOR PATTERN 
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It is shown that approaches other than the internal consistency 
method of estimating test reliability are either less satisfactory or 
lead to the same general results. The commonly attendant assump- 
tion of a single factor throughout the test items is challenged, how- 
ever. The consideration of a test made up of K sub-tests each com- 
posed of a different orthogonal factor disclosed that the assump- 
tion of a single factor produced an erroneous estimate of reliabil- 
ity with a ratio of (n—K)/(n—1) to the correct estimate. Special 
difficulties arising from this error in application of current tech- 
niques to short tests or to test batteries are discussed. Applica- 
tion of this same multi-factor concept to item-analysis discloses 
similar difficulties in that field. The item-test coefficient approaches 
V1/K as an upper limit rather than 1.00 and approaches V1/n as a 
lower limit rather than .00. This latter finding accounts for an 
over-estimation error in the Kuder-Richardson formula (8). A new 
method of isolating sub-tests based upon the item-test coefficient is 
proposed and tentatively outlined. Either this new method or a 
complete factor analysis is regarded as the only proper approach to 
the problem of test reliability, and the item-sub-test coefficient is 
similarly recommended as the proper approach for item analysis. 


The term reliability has been used loosely to apply to the resul- 
tant of the application of many different statistical operations. As 
a general rule these operations attempt to determine the verifiability 
of the original data and thus to establish the degree to which non- 
chance factors entered into the original measurements. 

One group of techniques approaches the problem through the 
direct comparison of the observed distribution of measurements with 
that which would have arisen by chance in accordance with certain 
theories of probability. These techniques parallel the use of the criti- 
cal ratio in establishing the reliability of means, sigmas, differences, 
and other statistical parameters by disproving the null hypothesis. 
One such approach is that of Jackson (7), who measures the sensi- 
tivity of the test, gamma (y), by taking the ratio of the standard 
deviation of the capacity to the standard deviation of chance. An- 
other writer, Hoyt (6), using the analysis of variance approach, sug- 
gests using the ratio of the true variance of the student responses to 
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the obtained variance among students. Edgerton and K. F. Thomson 
(3) suggest using the Lexis ratio to show that differences among stu- 
dents are greater than those among items. Hoyt shows that his re- 
sults are comparable to those of Jackson, and both Hoyt and Edger- 
ton and Thomson show their results to be comparable to those 
achieved by the use of certain of the Kuder-Richardson series [ac- 
tually to formulas (14) and (20), which involve only the sigmas of 
the items and the sigma of the total test] which we will discuss later 
at some length. Any restrictions demonstrated to hold for these par- 
ticular portions of the Kuder-Richardson series will thus apply to 
these probability methods as well. Allied techniques are (1) the 
Horst (4) maximized criterion which holds that the most reliable 
weighted composite is that with the largest relative variance, and (2) 
the item selection techniques based on the variance (difficulty) of the 
item. The remaining techniques attack the measurement of reliabil- 
ity through the verifiability of the original ranks of the members of 
the population. These techniques form two natural groups as the em- 
phasis on stability is (1) regardless of time, or (2) regardless of the 
particular test or items used. 

The first of these viewpoints—verifiability regardless of time— 
is exemplified by. the test-retest method of measuring reliability. 
While appearing best to meet the operations indicated in the usual 
definition of reliability, this method has been widely criticised from 
many different viewpoints. Typical criticisms refer to the effects of 
differential practice, memory, inability to duplicate testing conditions, 
inability to sustain motivation, etc. Perhaps even more serious is the 
objection raised that this coefficient is affected not only by the un- 
reliability of the test but also by the unreliability (lack of stability) 
of the function being tested. Paulsen (12) suggested measuring this 
trait fluctuation by correcting the test-retest coefficient for attenua- 
tion, using the split-half method for obtaining the reliabilities of the 
initial and final testings. Woodrow (18) suggested measuring this 
“quotidian variability” by the ratio of the actual sigma of the means 
of various samples to the sigma of the means as predicted from the 
average standard deviation of the multiple samples. Thouless (17) 
proposed the measurement of this “functional fluctuation” by what 
he called the double test-retest index, where he substituted alternate 
forms for the split-half approach of Paulsen. Thus in application we 
see that the “regardless of time criterion” in addition to many other 
ills requires an appeal to the preceding view—as in Woodrow—or to 
the remaining criterion of stability “regardless of test or items used” 
criterion—as in the split-half or comparable form methods of Paulsen 
or Thouless. Indirectly associated techniques are: (1) the original 
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Brown (1) concept as to the proper values for substitution in the 
Brown-Spearman formula; and (2) the concept of testing item reli- 
ability by means of individual changes in response on future test ad- 
minstrations. 

The remaining concept—the verifiability of the measurements re- 
gardless of the particular test used—is exemplified by the ‘‘compar- 
able test” method of measuring reliability. Here the procedure is 
complicated by two major difficulties: (1) the time element, and (2) 
the determination of comparability. With respect to time the argu- 
ment centers in the merit of separate alternate tests (assuming “com- 
parability” for the time being), with the necessary time separation 
involved, as compared to some split-half (odds vs. evens, first half- 
last half, etc.) technique (again assuming “comparability”) which 
would eliminate the time element. All of the objections raised against 
the test-retest method with the possible exception of the memory ele- 
ment would also apply against any time-separated alternate form ap- 
proach. Most recent writers have assumed the desirability of elimi- 
nating this time element by some method based on internal analysis. 

The second difficulty involved in this last viewpoint—the na- 
ture and determination of “comparability”—has usually been dealt 
with inadequately. It is always assumed but rarely demonstrated. 
The only really adequate definition of such comparability or equiva- 
lence known to the writers is that given by Kuder and Richardson (9) 
as their equation (1) which we hereby adopt. They say, 

“The correlation between two forms of a test is given by 


T (a+b+e+++-4n)(A+B+C+--4N) — 


Tas Fa FA — Tas Oa OB Pee + Tam On Ou 4 Tnn On On 





» (1) 
[oo? + on? + +++ + on? + 2 (Tap Ga oy + Tac Fade +++ + Tn om On) |? 
[o4? + og? + --- + on? + 2(145 04 o— + Tac 04 Oc +++ + Tun ou on) ]? 
in which a, b,---,” are items of the test, and A ,B,---, N are corre- 


sponding items in a second hypothetical test. Equivalence is now de- 
fined as interchangeability of items a and A , b and B, etc.; the mem- 
bers of each pair have the same difficulty and are correlated to the ex- 
tent of their respective reliabilities. The inter-item correlations of 
one test are the same as those in the other. These relationships con- 
stitute the operational definition of equivalence which is to be used.” 

As Kuder and Richardson point out, the above definition of com- 
parability makes the two terms in the bottom of equation (1) identi- 
cal, which reduces the formula for the true reliability of a test or test 
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battery with unit weights to the form 


Vee Gar + Top On + aida + Tan On? + 2 (Tar aq Ob = nisi ae Tnn om on) 





(2) 


A coe 
Oae + ap? + 2s + on? + 2 (Tan og oy + °** + Nmn Om On) 


If such a test as the capital letter series above were available it would 
truly be an alternate or comparable form. Actually such a test is sel- 
dom if ever available. Attempts at constructing alternate forms sel- 
dom hold rigorously to the above definitions. Instead it has been the 
custom to make either explicitly or implicitly the assumption of a 
single factor running through all such possible items and to construct 
alternate forms paying attention only to the difficulty of the items 
(equating means and sigmas, if even these are taken into account). 
The internal consistency hypothesis is the basis of the two most 
common methods of measuring reliability: (1) the split-half Spear- 
man-Brown (15) approach, and (2) the Kuder-Richardson series. 
Kuder and Richardson frankly assume a single factor among the test 
items, while the Spearman-Brown assumption of equal intercorrela- 
tions amounts to the same thing as is demonstrated when Kuder and 
Richardson derive the Spearman-Brown formula at one stage in their 
own series. Allied techniques are (1) the Edgerton-Kolbe (2) con- 
ception of the most reliable weighted criterion based on minimal dif- 
ferences among the scores of each individual for each of the criteria; 
(2) the Hotelling (5) conception of the most predictable criterion 
with weights proportional to the loadings on the first unrotated prin- 
ciple component; (3) and the many methods of item analysis, too 
numerous even to list, based on the item-test correlation coefficient 
or variants of that measure. When the assumption of a single factor 
is satisfied such procedures are justified, but frequently that assump- 
tion is not justified as is indicated by the many studies which show 
the usual test or test battery to contain many factors. What is the 
result of using these formulas when, as is probably almost universally 
the case, the basic assumption is not justified? Have we, perhaps 
been branding as unreliable tests which were satisfactory in that re- 
spect? Have we been discarding as unreliable items which were per- 
fectly good? The remainder of this paper considers these questions 
theoretically. Instead of the usual assumption of a single factor with 
equal intercorrelations and equal sigmas throughout the whole test, 
we shall instead assume a test of K factors—a, b,c, --- , k—each fac- 
tor being represented by a number of items n, which may vary from 
factor to factor. For the sake of simplicity and to maximize the dif- 
ference between this case and the usual assumptions these factors 
shall be taken as orthogonal or uncorrelated, i.e., the intercorrelations 
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between items in different factor groups are taken to be zero.* While 
the sigmas and inter-item correlations within a factor group may 
differ from group to group (1a # 7;;) and (0. # «;), we shall make the 
usual assumptions of equal sigmas and equal intercorrelations with- 
in each factor group (7, ge and «, = v) etc.) and shall 
further assume the reliability of each item ‘to be equal to the inter- 
item correlation holding for items in that group. Substituting these 
assumptions in equation (2) we see that the reliability of this mul- 
tiple-factor test would be 
Nz? a2" Ver 
it 2 . (3) 
Py Ng oe" a = Nz (Nz a 1) Oz" Tre 

We want to compare with this value, the value as estimated by the 
single factor theory. The Kuder-Richardson formula (14) is taken 
as the best measure of this type since it is this formula which the 
various direct probability approaches equalled and which Kuder and 
Richardson have shown to be equal to the Brown-Spearman method 
which is basic to split-half approaches to the measurement of reli- 
ability. We shall also see later on that this is the most general of 
their formulas which is applicable, since formula (8) of their series 
is erroneous. The Kuder-Richardson formula (14) reads 


— o;? a Py oz" i or)? 
Vtt er, (S* o;)? — Do? ; ss 


Substituting our present assumptions for our K-factored test in this 
equation gives 





(4) 





SE tte tte — 1) 00? Fae (S¥ te os)? (5) 


ltt — . 
KR (SE tte oc)? — DK te oe? DE Me oe? + DK Ne (Me — 1) 2? Toe 


and to bring out the difference between equation (3) and (5) more 
clearly we can rewrite (5) as 





* One of the editors objected to the stringency of this case, on the basis that 
it would seldom if ever occur in practice. We deliberately chose to magnify the 
discrepancy between our results and those based upon the assumption of a single 
factor, due to our feeling that the actual usual case would lie somewhere in be- 
tween the two extremes. The special case formulas derived in this paper were 
not meant to be used computationally but only to disclose the effects of various 
possible trends which might exist in practical situations. We do believe that the 
possible sub-tests and their inter-relations form the only sound approach to 
either the problem of test reliability or the problem of item-analysis. 
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bl Nz oc)? Ng? oe” Tow 


r. — 
ttxria (S¥ nz oc)? — SX nz 02? | DS Ne oe? + S¥* Ne (Me — 1) 02? Nee 


>* nN 27 (5a) 





Pg Nz oe" - Nz (Nz mal 1) or" Tog 


Now the first term in this bracket is the true value as given by equa- 
tion (3), which we see is reduced by an error factor (the second 
term in the bracket) and then increased again by a multiplier greater 
than one (the term preceding the bracket). We can see that the ex- 
tent of this error will depend upon the actual values of the n’s, o’s, 
and 7r’s. To evaluate the extent of this error under certain special 
conditions we can simplify the formulas by assuming various ones of 
these determiners, n,, o, and 7, to be equal from group to group. 
The following table gives the reduced form for 7;; and T:txr14 for each 
such possible set of assumptions. 


Several conclusions of interest and importance can be drawn on the 
basis of the equations in set (6). We note that: 


(a) The Kuder-Richardson formula equals the true formula 
only when K , the number of factors, equals one (Case g, since for 
the remaining single set the n’s, o’s and 7’s would all be equal, with 
K equal to one). With K equal to one, both formulas take the usual 
Brown-Spearman form, indicating that that formula is also correct 
for the usual assumption of a single factor. 

(b) The Kuder-Richardson formula tends to underestimate the 
true reliability by the ratio (n—K)/(n—1) when the number of fac- 
tors, K , is greater than one. (Cases d, g, and k.) 

(c) If every item in a test is perfectly reliable the test is per- 
fectly reliable even though all intercorrelations are equal to zero (n 
equals K and all #,’s equal one) and regardless of the size of the sig- 
mas (cases h, i, j, and k for the true 7;;) although the Kuder-Richard- 
son would not indicate this fact, giving values all of the way down to 
zero for extreme cases. 

(d) The size or uniformity of the sigmas is not important if the 
n’s and 7’s are equal (Case e) since they then drop out of the formula. 
This indicates that they are the least important of the three deter- 
miners. 

(e) The Brown-Spearman formula underestimates the true re- 
liability by the ratio of (n—K)/(n—1). (Case g.) The derivation 
of this formula is simple and easily understood. The usual Brown- 
Spearman formula reads 
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se N Veo 
BS 1+ (n—1) Tee 





(7) 


Vet 


where r,, equals the average intercorrelation for all the items, but for 
any given item under the condition of equal n’s, o’s, and 7’s for each 
factor group the average intercorrelation would be 


"Reet | co + “—~ We 
awe a (n and (8) 
n-—1 





and since n, would equal n/K we would have 


n—-K fa (9) 
Tee = ’ 
K n-1 





and substitution of this value in equation (7) yields 
Bess K N Toa 
» 6643 B+ (n — K)T oa 





(6, g) 


Tet 


While this limiting error of n—K/n—1 for the two internal consis- 
tency methods of estimating reliability [see conclusions (b) and (e)] 
becomes negligible when n becomes very large, it is nevertheless true 
that in short tests or when the formulas are used to estimate the re- 
liability of test batteries where the number of tests is usually small, 
the equations based on internal consistency (assumption of one fac- 
tor) would lead to gross underestimation and serious theoretical dif- 
ficulty. 

As to the gross underestimation, we present two cases of 12-item 
tests or batteries with assumed intercorrelations within the factor 
groups (the reliability of each item) equal to .95 for one test and to 
.60 in the other. For each test we shall assume anywhere from one 
to twelve factors with from twelve to one items, respectively. The 
true and estimated values follow: 


Toa = .95 Toa = -60 
K Nz Tee Yetxria Tee TrexRia 
az 1 .950 .000 .600 .000 
6 2 .974 531 -750 .409 
4 3 .983 715 .818 595 
3 4 .987 .808 .857 -701 
2 6 991 901 .900 818 
1 12 .996 .996 947 947 
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An example of attendant theoretical difficulty due to the erroneous 
use of the internal consistency theory of reliability is the supposed 
dilemma cited by Royer (14) for the problem of multiple correlation. 
Low intercorrelations are a prerequisite to achieving high validity, 
whereas the consistency hypothesis holds that low intercorrelations 
mean low reliability. This of course does not follow at all if we make 
the probably nearer correct assumption of multiple orthogonal fac- 
tors in both the criterion and the test items. It is of course true, as 
Thompson (16) has shown, using the correct reliability formula, that 
maximum validity and maximum reliability are not identical goals. 
If the criterion happens to have its variance determined ‘in large part 
by factors predominating among the less reliable portions of the pos- 
sible predictor items, then maximum validity will mean greatly low- 
ered reliability as compared with the prediction of a different cri- 
terion whose factor composition is proportional to the reliabilities of 
the selected items, i.e., with a comparable: test. We cannot follow 
Thompson’s idea of compromising the weights in this case, however, 
since getting a better measure of something a person does not want 
to measure—a more comparable test—is no advantage if one thereby 
secures a poorer measure of what he is trying to measure—the cri- 
terion. The proper mode of approach in the Thompson situation 
would be to improve the reliability, by lengthening or other means,, 
of those portions of the test battery with high regression weights, 
thereby increasing validity as well as reliability, rather than the 
Thompson suggestion of shifting the weights so as to secure higher 
reliability at the expense of decreased validity. 

One other serious consequence of the erroneous adoption of the 
single factor assumption is a widespread misuse of the item-test corre- 
lation coefficient as a method of item selection and elimination. It 
is true that this measure, as Richardson (13) has clearly demonstrat- 
ed, is the appropriate measure to use when there is a single factor 
among the items, but let us examine the values of this coefficient for 
various items under the condition of multiple orthogonal factors as- 
sumed above. In general this coefficient for any given item will equal 





Tit =o;[1 a ie) (nj — 1)rii]/V> Ny, oz" + > Nz (Nz It: 1) oz" Voz 5 (10) 


where the subscript 7 stands for the particular group of items having 
the same factor as the item in question. We see then that the item- 
test coefficient depends not on reliability (7;;) alone but also upon o; 
(item difficulty) and upon the number of items in the test measuring 
the same factor (n;). Assuming these various determiners to be 
equal from group to group gives a series of values paralleling the 
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various conditions in set (6) above: 
Equal Unequal Tit Ches 
4 aiasil oiLK + (n— K)ri] m 
Vn VK Sof + (n— K)S 2 te 
1+ (m — 1) ri 
Vn +X Me (Me — 1) Tae 
oi{1 + (m; — 1)7i] 





















































r o,n Cc 
VD Ne oe? + Fit Zz. Metts — 1) 0,” (11) 
K+ (n— K)rii 
N,o : as ee d 
Vn VK? + (nun — APS Voe 
, K+ (n — K)7rii 
n,7 o Ci e 
a> of 
o,? n 1+ (n; —1)%i f 
V (1 — 75)" + DS 2? 
\— (n — K) fii 
47 5@ — gz 
Kn 
r=1.00 n,o ms Fon h 
VD 1,2 of 
r=1.00,n o oi V1/3 a,’ i 
r=1.00,¢ n ni/V> 0? j 
r=1.00,c,n — Vi/K k 


From these equations in set (11) we can draw the following conclu- 
sions: 
(a) The item-test coefficient is not a measure of item reliability 
alone, but depends upon the share of the total variance of the 


battery determined by the sub-battery of which it is a part. 
(b) Perfectly reliable items with low n’s and o’s would be dis- 
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carded as worthless by the usual method of applying this cri- 
terion, while less reliable items with large n’s and o’s would be 
retained. Items with the highest 7;, value are not necessarily the 
best items. 

(c) The value of 7;; does not approach 1.00 as an upper limit as 
the items become more reliable, but instead, (”’s and o’s being 
equal) approaches the value of \/1/K. Of course if K =1, ri: ~ 
does approach 1.00, but if K is greater than one the possible 
upper limits are 


ii — 1.00 73; = 50 
Case k Case g 

K upper limit of 7;; 

2 ie 505 

4 .500 .361 

6 408 297 

8 354 .260 
10 316 .234 
20 224 173 


The automatic rejection of items with low ri; values is not jus- 
tified. 


(d) If 7r;; is equal to zero, the lower limit of the value of 7;; ap- 
proaches V1/n rather than zero, which makes the value of the 
Kuder-Richardson formula (8) (their article) fictitiously high. 
According to their concept of the single factor their formula (8) 
should, as their formula (14) does, become equal to zero when 
all of the intercorrelations are equal to zero, but this is not the 
case. Their formula (8) is 


of ~ Der Leare seo - Bel’ 
Tt = + -+- . (12) 











KR8 2 or? ao? » O17 


which if we assume one factor with all o’s equal becomes 


o;7 — Nn es. oi? > eT 4 of — No;* « 
r = 
tteRs A o;? or? 2 o:? ° (13) 


If we let all 7.. values be 1.00, we have 








ofp =n oi and Cit = 1.00 , 


whence 
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noti—n oi? N a3? nN? o;? — nai? . (14) 
*~,,, + - = 1.00, 


KR8 2 Nn? oi? n? oi? 2 n? oi? 











which is the correct answer. But if we let all 7,, values equal .00 








we have 
oe? — Noi? and tip =1/Vn 
whence 
(15) 
N oi? — No;? o;? N o;? — No;* |? en 
Pig SE erie + = V1/n > .00, 
KR8 2n oi? N o; 2n oi? 


which is incorrect. While this is the upper limit of error in this 
formula, it does remain spuriously large for all values of 722 less 
than 1.00. The source of this spurious increment is the incorrect 
assumption of a self-correlation of 1.00 for 7;; in the item-test 
coefficient. While this is correct for the actual correlation with 
the test of which it is a part, the substitution of this value in the 
Kuder-Richardson series assumes it to hold as well for the “‘com- 
parable” test series, the capital letter test, whereas the correct 
value here is not 1.00 but the actual reliability of the item. Thus 
when all 7,, values are unity the reliability is 1.00 and formula 
(8) is correct, but for all other values of 7,, the reliability will 
be less than unity and formula (8) will be in error, this error 
reaching a maximum @as 7, approaches zero. 


Some numerical examples of typical solutions for 3-factor prob- 
lems are presented here as an indication of the points made under 
conclusions (a), (b), and (c) above: 


Case n o ig Tix Case n o Y Tit 
; 50 8 .90 .308 50 5 .20 105 
a 50 A .70 .326 d 50 5 .70 .303 
50 5 .20 141 50 5 .90 383 

100 A .20 .362 50 5 .60 .559 

b 40 A -70 492 e 50 4 .60 447 
10 A .90 .158 50 3 .60 .335 

100 5 .60 .362 100 5 .60 534 

Cc 40 A .60 492 f 40 5 .60 .216 
10 2 .60 .158 10 5 .60 .057 
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Case n o T T;, Case n o r Tit 
100 5 41.00 .940 100 4 1.00 .887 
h 40 4 1.00 3204 = j 40 4 1.00 .355 
10 3 1.00 .057 10 4 1.00 .089 
50 5 1.00 421 
i 50 4 1.00 .oo7 
50 8 100 .253 


In view of the general possible unsatisfactory condition resuit- 
ing from the application of the present methods of estimating reli- 
ability of the total test and of item validation by the blind assump- 
tion of a single factor, it seems advisable to suggest a marked revi- 
sion in the present methods of test analysis. The obvious solution 
would be the factorial analysis of each test, but while ideal this would 
be very laborious and often impracticable, especially if the number of 
items were at all large. The calculation of the intercorrelations would 
alone be a tremendous undertaking to say nothing of securing the 
residual matrices and the final rotation of the obtained loadings. Let 
us consider simpler possibilities. 

A start toward such a method is found in the concept of “item 
synonymization” advanced by Lentz and Whitmer (10). In their 
method item intercorrelations have to be computed and there are no 
clear-cut standards for inclusion of an item in any given synonymy. 
However, they have demonstrated that 


(a) test items do tend to fall into groups, 


(b) an item correlates more highly with its synonymy 
than with other synonymies, and 


(c) synonymies tend to correlate lowly with each other. 


If these synonymies were at all numerous in a test a much better esti- 
mate of its over-all reliability could be obtained by considering the 
synonymies as sub-tests, computing their reliabilities by the Kuder- 
Richardson formula( justified for such a consistent group), and then 
computing the total test reliability by using these coefficients to- 
gether with the inter-synonymy correlations in their general formu- 
la (2). Item validity would be evaluated not in terms of the corre- 
lation with the total test put in terms of the item correlation with 
any of the: synonymies. 

The present writers suggest tentatively an approach based on 
item-test rather than inter-item correlation coefficients. The follow- 
ing steps constitute a job-analysis of the proposed method: 
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(1) Compute the total test score (7) using unit weights. 
(2) Compute all item-test correlations (7:7). 
(3) Skim off the items with the highest such coefficients. 
(4) Rescore the papers on the basis of the selected items 
(S;). 
(5) Compute all item correlations with the new score 
(7is,) . 
(6) Add new items where ris, > rir and drop items where 
Tis, < Tir to form S7. 
(7) Rescore the papers on the basis of the amended list 
(S,'). 
(8) Repeat steps (5) and (6), computing 7is,’, adding 
items where ris) > Yis, and dropping them when 
Tis, < Tis, e 
(9) Compute S,” and repeat steps (7), (8), and (9) until 
no further changes are indicated, computing S,”, S;'"’, 
oo eM, 
(10) Record the final 7;s," values and compute a new resi- 
dual score 7” = T — S;". 
(11) Repeat steps (2) through (9) using 7” in place of T 
and S,;* in place of S/*. 
(12) Record the final ris," values and compute a new resi- 
dual test score T” = T’ — S,,” and continue as before. 
(18) Repeat the entire process through 7” until the test 
items are exhausted or until all 7:7" values approxi- 
mate zero. 


The selected sub-tests S,", Su”, Si”, etc., will correspond to 
item synonymies of Lentz, will tend to lie along pre-rotated orthogon- 
al axes (if the 7 items possessed simple structure to begin with), and 
the ris,” values will be the factor loadings of the individual items on 


those axes. 
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We can illustrate this method of taking the three-factor numeri- 


100 values of .534, 
40 values of .216, 
and 10 values of .057. 


cal examples given for case (f) in the discussion of 7;;, above. Here 
the correlations of the items with the total test would be 
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We would select the 100 highest items as S,, rescore the tests, and 
compute the Tis, values, obtaining: 


100 values of .602 ,* 
and 50 values of .000, 


completing that phase. We would next compute T — S, = T’ and com- 
pute r,,,, obtaining ° 
100 values of .000, 
40 values of .589, 
and 10 values of .155. 


We would then form S. from the 40 items with correlations of .589, 
rescore the papers, and then compute ris,, obtaining 


110 values of .000, 
and 40 values of .607 ,* 


completing the second phase. We would next compute T” = 7’ — S, 
and compute r,,,,, obtaining 


140 values of .000, 
10 values of .627 .* 


Using these 10 values to form S;, since rescoring would result 
in identical values, would complete the analysis of the test 

The Lentz technique would have arrived at identical sub-tests 
but at a cost of 150 X 149/2 = 11,175 correlation coefficients rather 
than the present 6 X 150 = 900 sub-calculations. Of course all tests 
would not possess simple structure nor would they all contain an hier- 
archical arrangement such that n,, > %., > Ms,, etc. In case the sev- 
eral n’s or the larger of these approached equality or in case the con- 
tribution to the variance approached equality the separation of the 
factors would become more laborious and perhaps impossible. 

Mosier (11) has warned against the use of ri; in a two-factor 
test when (1) the factors are numerically equal, and (2) the items 
do not possess simple structure, i.e., have loadings on both factors. His 
general thesis is in line with the main argument of this paper, which 
extends the consideration to more than two factors and applies the 
idea to the whole problem of reliability as well as to item analysis. 
That his warning is pertinent as a criticism or possible limitation to 
the method of test analysis proposed above is also recognized. 


; * Computing the correlation of the item with the total not counting the item 
in question would produce the true value of .60 here and in later synonymies, but 
such error will not obscure the major relationships. 
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In two attempts to use this method empirically the writers found 
one test which worked out quite smoothly due to unequally represent- 
ed factors, whereas in a second test the two main factors were equi- 
potent and came out as a single sub-test until S; was broken up into 
Siz and S;z by visual inspection, after which the method proceeded to 
work satisfactorily. 

The present writers intend to examine further the application of 
the suggested method of analysis to both theoretical and empirical 
tests. If it can be made to overcome the difficulties involved in equi- 
potent factors it may well provide a practicable means of approximate 
factorial analysis for test situations where the usual methods would 
be prohibitive. The results of such research will be published in a 
later paper. 

An article by Kelley (8), published after the beginning of the 
present article, must be given special mention since it foreshadowed 
empirically a number of the rational conclusions presented above. His 
finding that formulas (14), (20), and (21) of the Kuder-Richardson 
series yielded, for a three-item test with inter-correlations of zero, 
values of .00 for the reliability coefficient, while their formula (8) 
yielded a value of .58, is a perfect example of our equation (15) show- 
ing the erroneous nature of their formula (8) and of the incorrect- 
ness of their general approach when the number of factors, K, is 
large in comparison with the number of items, n. Since the Kelley 
article did not point out the incorrectness of their formula (8) and 
especially since he repeated their claim that it was their most reli- 
able equation, it was feared that his reporting of the value .58 might 
be taken as lending support to that spurious equation. 

While we agree that the Kuder-Richardson series is a measure 
of coherence rather than of reliability, we cannot accept Kelley’s co- 
efficient of coherence, VC/SVC, since it gave a value of .33 for a 
test whose coherence is obviously zero. The reason for this errone- 
ous result is the same as that responsible for the error in the Kuder- 
Richardson formula (8). They used unity for the item reliabilities 
while the Kelley method of factor analysis is equivalent to putting 
these same fictitious unities in the diagonals of the factor matrix. Had 
Kelley used the correct communalities of zero, i.e., had he used the 
Thurstone approach, he would have attained the correct coefficient of 
coherence of .00 as yielded by the Kuder-Richardson equations (14), 
(20), and (21). The fact that he obtained .33 while the Kuder-Rich- 


ardson formula (8) yielded \/.33 or .58 is empirical evidence of the 


similarity of error. 
We also cannot agree to Kelley’s appeal to an “act of judgment” 
on the part of the experimenter, as in splitting a test in half or other 
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smaller fractions, as a valid or dependable method of computing reli- 
ability. The suggestion of a complete factor analysis as a basis for 
setting up sub-tests or the short sub-test selection method schema- 
tized above (if it proves practicable) with the then proper use of the 
basic Kuder-Richardson equation appears to be a sounder and much 
more scientific approach. 
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A NOMOGRAPH FOR RAPID DETERMINATION OF MEDIANS 


CLIFFORD E. JURGENSEN 
KIMBERLY-CLARK CORPORATION, NEENAH, WISCONSIN 


Directions are given for constructing a very simple nomograph 
for computing medians, which is entered with information from the 
cumulative frequency distribution. It gives a linear interpolation 
within the class interval containing the median. 


Computations of medians and semi-interquartile ranges are gen- 
erally considered to be quickly and easily made. At times, and with . 
regard to some types of work, such computations are so numerous, 
and consequently so time consuming, that the proposed project hardly 
seems worth the time required. This is perhaps especially true in the 
industrial situation where there is a critical shortage of clerical help 
and where expansion and turmoil resulting from the war situation 
have made the utilization of scientific techniques even more desirable 
than previously. 

Such a situation confronted the author recently when developing 
an employee merit rating scale consisting of scaled statements which 
were to be checked by supervisors as applying or not applying to the 
employee being rated. A total of 352 statements indicative of em- 
ployee merit were compiled, and each statement was given a rating 
which ranged from one to nine inclusive. One hundred supervisors 
rated each of the statements. In order to determine the scale value of 
each statement it was necessary to compute 352 medians, each being 
based on 100 cases. To determine whether supervisors agreed with 
each other regarding the value of each statement, and consequently 
whether or not it was usable in the final check list, it was necessary to 
compute 352 semi-interquartile ranges. For each of these, the first 
and third quartile had to be computed. Thus a total of 1,056 medians 
and quartiles had to be computed. Inasmuch as computations were 
being made by clerks relatively untrained in statistical procedures, it 
was decided to make each computation twice in order to assure ac- 
curacy, thus making a total of 2,112 required computations. Persons 
making the computations originally followed the usual formula of 


I -F i 
Median = 1 + ; : 
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A similar formula, counting from the upper end of the distribution, 
was used as a check. The same formulas, changing the N/2, were 
used for the quartiles. 

Shortly after the statistical computations had been started, it be- 
came obvious to the author that considerably more time would be re- 
quired than had been anticipated. In the search for a shortcut, a 
graphical means for determining medians and quartiles was devised. 
The procedure is so simple that it is presented with apologies, al- 
though neither the author nor other psychologists with whom he has 
discussed it have seen or heard of the procedure being used previously. 
It has been found to give results identical with the arithmetical meth- 
od, although more accurate because fewer errors were made than in 
the arithmetical method. No comparative record of time required was 
kept, but it is estimated that the graphical procedure required less 
than one-fourth the time necessary for the arithmetical method. 

The nomograph was constructed merely by obtaining a large 
sheet of heavy quarter-inch cross-ruled paper and drawing two paral- 
lel lines 25 inches in length and ten inches apart. A cross bar was 
drawn halfway between the ends of each line, thus making a large H. 
The 25-inch length of the vertical lines allowed one quarter inch for 
each of the 100 cases in the frequency distribution. The horizontal 
line was divided into one-inch lengths, each inch representing one 
tenth of a point. (See Fig. 1 for reproduction in reduced size.) 

The nomograph is used as follows: (1) the cumulative frequency 
of the distribution is obtained; (2) a rule is placed on the left vertical 
line at the point representing the cumulative frequency immediately 
below N/2; (3) the rule is pivoted on the left vertical line so that the 
right vertical line is crossed at the point representing the cumulative 
frequency immediately above N/2; (4) the number of tenths indi- 
cated by the point at which the rule crosses the horizontal line is 
added to the lower end of the interval in which the median falls. The 
resultant median is correct to the first decimal place, and may be esti- 
mated (if desired) to the second decimal place. 

Original ratings of statements for which this nomograph was 
specifically devised, were made on a nine-point scale. Each point was 
considered to represent the range from that point plus or minus .5. 
In order to eliminate the necessity for adding tenths to a decimal 
number (e.g., .8 + 2.5), the one-inch sections on the cross bar were 
each increased in size by .5. The reading on the horizontal line was 
thus added directly to the midpoint score of the interval containing 
the cumulative frequency immediately below N/2. Inasmuch as me- 
dians were to be computed only to the first decimal place, sections on 
the horizontal line were marked to indicate the range included in 
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each decimal number, e.g., the section marked .7 covers the range 
from .65 to .75. 

First and third quartiles are computed in the same way as me- 
dians, the horizontal lines being drawn at N/4 and 3N/4 respectively, 
these figures also being used in determining the cumulative frequen- 
cies at which each vertical line is entered. Obviously, any other per- 
centile can be computed similarly. 

Following is a specific example of the use of the nomograph (Fig. 


a}: 


Ratings 1 2 3 4 5 6 7 8 9 
Frequencies 1 8 23 46 I17 4 1 
Cumulative 


Frequency 1 9 32 78 95 99 100 
To compute median: 


1. Enter left vertical line at 32 (cumulative fre- 
quency just below N/2) 

2. Enter right vertical line at 78 (c.f. just above 
N/2) 

3. Read the tenths indicated -where rule crosses 
horizontal line (.9) and add to midpoint of in- 
terval containing cumulative frequency just be- 
low N/2 (38.0) to secure median (3.9). 


To compute first quartile: 


1. Enter left vertical line at 9 (cf. just below 
N/4) 

2. Enter right line at 32 (c.f. just above N/4) 

8. Read the tenths on the lower horizontal line 
(1.2) and add to midpoint of interval contain- 
ing c.f. just below N/4 (2.0) to secure Q, (3.2). 


To compute third quartile: 


1. Enter left vertical line at 32 (c.f. just below 
3N/4) 

2. Enter right vertical line at 78 (c.f. just below 
3N/4) 

3. Read the tenths on upper horizontal line (1.4) 
and add to midpoint of interval containing c.f. 
just below 3N/4 (3.0) to secure Q; (4.4). 


As will be noted from the examples above, the points of entering 
the vertical lines for the quartiles are sometimes the same as for the 








268 PSYCHOMETRIKA 


median, with this example being the same for Q; as for the median. 
Considerable time can be saved if the user of the nomograph watches 
for such similarity. 

The nomograph in Fig. 1 is published primarily for illustrative 
purposes and can be used only when N equals 100 or when N is ex- 
pressed in terms of percentage. A similar nomograph can be con- 
structed for any size N in less than ten minutes time. Cross-ruled 
paper should be used, and to insure accuracy each space on the verti- 
cal line should represent one individual. The distance between the 
vertical lines is immaterial provided the space can conveniently and 
accurately be divided into ten equal parts for results accurate to the 
first decimal place. Where desired, the distance between vertical lines 
can be divided into 100 spaces to secure accuracy correct to the sec- 
ond decimal place. Using eighth-inch cross-section paper, the dis- 
tance would be only twelve and one half inches. Large nomographs 
of such size are easily read with great accuracy, and thus are superior 
to smaller reproductions such as illustrated here. 
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